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PREAMBLE 


In March 2010 the Standing Committee on Public Accounts held public hearings 
on the Education Quality and Accountability Office (the Office or Agency), the 
subject of an audit by the Auditor General in 2009.' Ministry of Education 
witnesses appearing before the Committee included the Deputy Minister, 
Assistant Deputy Minister and senior staff. Witnesses from the Office included 
the chair, the CEO and others.” This report highlights the Auditor’s observations 
and recommendations contained in Sec. 3.04 of his 2009 Annual Report and 
presents the Committee’s own findings, views, and recommendations. 


Acknowledgements 


The Standing Committee endorses the Auditor’s findings and recommendations. 
It also thanks the Auditor and his team for drawing attention to the important 
issues stemming from the EQAO process in Ontario. Finally, the Committee 
would like to acknowledge the assistance provided during the hearings and report 
writing by the Office of the Auditor General, the Clerk of the Committee, and 
staff of the Legislative Research Service. 


OVERVIEW 


Objectives and Scope of the Audit 


The audit objective assessed whether the Education Quality and Accountability 
Office had adequate systems, processes, and procedures in place to ensure that: 


« student assessment results were comparable from year to year and accurately 
reflected student performance in regard to the Ontario curriculum; 


« legislative and policy requirements were being fulfilled; and 


« goods and services were acquired and programs delivered in an economic and 
efficient manner. 


The scope of the audit included research on student assessment practices in other 
jurisdictions, review and analysis of EQAO administrative directives, policies, 
and procedures, as well as interviews with agency board members and staff, 
including two psychometric experts. Other interviews included personnel from 
four school boards—Peel District, Halton District, Hastings and Prince Edward 
District, and Peterborough Victoria Northumberland and Clarington Catholic 
District; and personnel from the Elementary Teachers’ Federation of Ontario, the 


' See Section 3.04 of Ontario, Office of the Auditor General, 2009 Annual Repo (Toronto: The 

Office, 2009), pp. 128-142, Internet site at 
http://www.auditor.on.ca/en/reports_en/en09/304en09.pdf accessed on September 7, 2010. 

* For a transcript of proceedings, see Ontario, Legislative Assembly, Standing Committee on 

Public Accounts, Hansard: Official Report of Debates, 39" Parliament, 2™ Session (31 March 


2010), Internet site at http://www.ontla.on.ca/committee-proceedings/transcripts/files_pdf/31- 
MAR-2010 P002.pdf accessed on September 7, 2010. 


Ontario English Catholic Teachers’ Association, and the Council of Ontario 
Directors of Education. 


Background 
Agency and Mandate 


The Office was established in 1996 following upon a recommendation of 
Ontario’s Royal Commission on Learning. After extensive consultations with 
teachers, parents, students, and the public, the Commission concluded that 
province-wide testing of students was necessary to monitor student achievement 
and respond to the public’s demand for clarity and greater accountability with 
regard to student achievement. 


Created as an independent operational service agency to conduct province-wide 
assessments of students, the Office has administered about 600,000 tests annually 
since 2000/01. It employs approximately 140 permanent staff complemented by 
approximately 1,700 seconded or temporary staff during marking periods. It spent 
$31.7 million in the 2008/09 fiscal year, all of it funded by the Ministry. Agency 
officials note that the cost of the EQAO averages about $15 per student out ofa 
$20 billion dollar education budget for elementary and secondary education in 
Ontario.” 


The Office is mandated to develop, administer, mark, and report on province-wide 
tests of student achievement. The results are intended to provide reliable, 
objective, and high-quality data that can be used as a tool by the Ministry of 
Education (the Ministry) and the province’s 72 school boards to enhance student 
learning and improvement planning. 


The Office develops test questions based on the Ministry’s school curriculum 
expectations and designed to provide an objective appraisal of student 
achievement. It aims to ensure that tests have a similar level of difficulty from one 
year to the next so that results can be compared over time. The Agency is 
responsible for providing specific guidelines for school boards, principals and 
teachers to follow in delivering the tests to students. It oversees test scoring, and 
must publicly report test results and make recommendations to the Ministry. 
These pertain to any matter related to the quality or effectiveness of elementary 
and secondary education in Ontario. 


Since the tabling of the Auditor’s 2009 Annual Report, Agency officials have 
taken steps to address all of the relevant recommendations and in some cases have 
already implemented some of the actions required. Officials take the view that 
the EQAO program of assessments, which focuses on measuring the achievement 
of every student, has been a catalyst for improvement of thousands of students 
across Ontario. 


* Ibid., p. 33. 


Assessments 


Each year there is an EQAO assessment of students in all Ontario publicly funded 
schools in Grades 3, 6, 9, and 10. Grade 3 and 6 students are tested in reading, 
writing, and mathematics while Grade 9 students are tested only in mathematics 
(applied and academic). As a condition of high school graduation, all students, 
including those in private schools, must pass the Ontario Secondary School 
Literacy Test (OSSLT). The OSSLT is usually written in Grade 10 and it © 
determines if students are meeting the minimum standards for literacy. Provincial 
EQAO assessments provide Ontarians with a snapshot of student achievement at a 
particular point in time. The assessments developed by EQAO must include five 
different assessments in both French and English. The agency prints, delivers, 
administers, collects, marks, and reports the assessments given at approximately 
4,300 schools across Ontario. 


Legislation and Guiding Instruments 


The Agency’s governing instruments are legislation (the Education Quality and 
Accountability Office Act), the Agency Establishment and Accountability 
Directive, and a Memorandum of Understanding. 


ISSUES RAISED IN THE AUDIT AND BEFORE THE COMMITTEE 


Significant issues were raised by the audit, and before the Committee. The 
Committee attaches particular importance to those issues discussed below. 


Test Development and Administration—Ensuring Consistency 


The Auditor found that the Agency had adequate procedures and controls for 
ensuring that its tests accurately reflected the Ministry’s curriculum expectations. 
To ensure comparability of the tests’ level of difficulty between years, the Agency 
had imposed strict criteria for the development and field testing of questions, and 
thoroughly reviewed test content. It was observed that the Agency employs a 
number of quality assurance measures to ensure consistency in the level of test 
difficulty from one year to the next. 


The audit team interviewed two of the psychometric experts who advise the 
EQAO on matters related to testing models and the more technical aspects of 
assessments known as Item Response Theory (IRT). IRT provides a framework 
for evaluating how well an assessment works and how well it measures student 
achievement by allowing comparisons of assessment results over time. Both 
experts agreed that the EQAO process is thorough and ensures consistency from 
one year to the next. 


Ministry officials indicated to the Committee the progress that public education is 
making in the province. For example, more students are graduating from high 
school. In the past five years, the graduation rate has risen from 68% to 79%. In 
addition, literacy and numeracy skills have steadily improved. According to the 
Ministry, EQAO results show that in 2003-04, 54% of students were achieving at 
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or above the provincial standard.’ In 2008-09, 67% of Grade 3 and Grade 6 
students were achieving that standard—a gain of 13 percentage points since the 
earlier date.° 


During the public hearings, Members commented on that same observation. The 
rising EQAO assessment scores suggest that student achievement is on the rise. 
Yet, Members are hearing anecdotally from university professors that the quality 
of students entering university appears to be lacking. Members were told by 
faculty that university students cannot properly write sentences. These comments 
do not match what Members are hearing about rising assessment results. 

Some Members asked if the EQAO scores might be rising because—in addition to 
all the good work that the Ministry’s achievement division is doing—teachers are ~ 
teaching to the test. 


Agency officials clarified that the test reflects measures of student achievement in 
relation to the Ontario curriculum. Moreover, when the EQAO assessments were 
first administered in Ontario, the curriculum was quite new. Today, teachers are 
much more familiar with the curriculum which might account for the rising 
Scores. 


While Agency officials could not comment on the concerns of university teaching 
staff about the quality of students entering university, they referenced a recent 
study of mathematics. It suggested that about 58% of students arriving at 
university had a mastery of mathematics. Officials also observed that a similar 
percentage of students leaving Grade 6 are also judged to be competent in 
mathematics. Another Agency official, and former college president, noted that 
historically universities have faced the challenge of accepting students who have 
moved through the education system without the required competencies. On the 
other hand, the prospect of remediation of competency gaps is significantly 
enhanced through early identification through EQAO assessments. 


Members also asked officials about the makeup of the tests to ensure year to year 
-comparability. They wondered if the Office had the capability of analyzing the 
past scores of only the multiple choice component of the assessments. Some 
Members queried if an analysis of the results of multiple choice questions—which 
are less subjective than open response questions—might yield a steady rise of 
EQAO scores. Agency officials responded that they undertake an established 
statistical procedure for equating from year to year which includes both multiple 
choice and open response items.’ Although the Agency has not analyzed just the 


* The provincial standard is Level 3 which roughly corresponds to a B. 
° Standing Committee on Public Accounts, Hansard, p. 21. 
° Ibid., pp. 28-29. The Student Achievement Division of the Ontario Ministry of Education is 
headed up by an assistant deputy minister who works closely with the Education Quality and 
Accountability Office. During the public hearings she explained the types of interventions and 
strategies utilized at schools and with school boards and individuals to improve student 
achievement. For examples see pp. 29, 32, 34-35, 36-37. 

The purpose of equating is to ensure that valid comparisons of test results over time can be made. 
The Education Quality.and Accountability Office aims to construct assessment forms of equal 


multiple choice items in a systematic way, it would be possible to do. However, 
they also cautioned that one does not usually see large differences by analyzing 
only multiple choice items. Officials expressed confidence that if they evaluated 
only multiple-choice items one would see a similar pattern across the years. 


The Chair of the Standing Committee has written to officials of the Education 
Quality and Accountability Office seeking supplementary data analysis on the 
testing results for Grades 3 and 6—reading, writing and mathematics—and the 
Grade 9 Assessment of Mathematics (applied and academic), as well as the Grade 
10 Ontario Secondary School Literary Test (OSSLT). 


Exempted Students 


While all students are expected to write the EQAO assessments, school principals 
may exempt those with special needs as well as students for whom Englishisa 
second language. Public reporting of the overall scores on EQAO tests include 
exempt students, but such students are counted as not having achieved the 
provincial standard in the assessment scores. Thus, schools with a 
disproportionately high number of exempt students would receive lower overall 
scores than otherwise comparable schools. Teachers and principals claimed that 
this policy distorts EQAO reported results. 


The Auditor recommended that the Office assess the equity of including exempt 
students in the overall assessment results as having not met the provincial 
standard. He also recommended that schools and school boards where the 
number of exempt students appears to be relatively high be followed up by the 
Office to ensure that the exemptions are justified. 


In response the Ministry noted that principals, together with parents, make the 
determination about which students are unable to write the assessment even with 
accommodations or special provisions. If the Office were to exclude exempted 
students when reporting a school’s results, schools that work to ensure that all 
students are included would view the Office’s practice as inequitable. In the 
Ministry’s view, it is important that all students have the opportunity to 
demonstrate their achievements. 


Public Hearings 


Ministry officials informed the Committee that when looking at the achievement 
levels of Ontario students, the Ministry has successfully reduced the number of 
students exempted from the tests over the last number of years. Members asked 





difficulty each year, but its assessments might differ slightly in difficulty from year to year. The 
equating process adjusts for such differences. It ensures that the skills and knowledge required for 
students to be classified at Level 3 are equivalent from year to year and that changes in 
achievement results across years are due to differences in students’ knowledge and skills and not 
to differences in test difficulty. See Michael Koslow, Director, Data and Support Services, “A 
Comparison of Four Test-Equating Methods,” EQAO Research Bulletin #3 (January 2010), p. 1, 


Internet site at http://www.eqao.com/Research/pdf/E/ResearchBulletinCrb3 ne 0110 web.pdf 
accessed on October 4, 2010. 


the witnesses about the types of students who would likely be exempted from the 
EQAO assessments. | 


Agency officials responded that generally, a principal may exempt students with 
special education designations as well as newcomers. On the other hand, many 
special education students do participate in the EQAO assessments. The general 
guideline that principals adhere to is this: if the student is following the Ontario 
curriculum, there is little justification for exempting such students from the 
assessment, which is based on the Ontario curriculum. Moreover, in no case 
should a student be exempted without the parent being consulted. 


The Agency’s CEO noted that she has had conversations with the Minister’s 
Advisory Council on Special Education and the Learning Disabilities Association 
- of Ontario. Both have strongly advocated that special needs students be included 
in the assessments because parents want to know how their children are achieving 
relative to the curriculum. 


Committee Recommendations 
The Standing Committee on Public Accounts recommends that: 


1. On the matter of principals exempting students from EQAO 
assessments for acceptable reasons and then assessing such students 
as though they did not achieve the provincial standard, the Standing 
Committee asks that the Ministry of Education and the Education 
Quality and Accountability Office report to the Standing Committee 
as to whether they have considered other options, in particular, for 
those schools where a disproportionately large number of students 
have been exempted. 


2. The Education Quality and Accountability Office report to the 
Standing Committee the numbers of exempted students over the past 
five years with an explanation for any trends that have occurred over 
that timeframe. | 


Quality Assurance 


Many teachers and principals commented to the audit team that as EQAO results 
take on broader acceptance, there is ever-increasing pressure to improve results. 
The EQAO assessments carry some risk of irregularities or non-compliance with 
administrative procedures. In the assessments involving the lower grades (Grades 
3 and 6) the primary risk involves teachers or principals. For instance, teachers 
must not do or say anything to influence students to alter their responses during 
testing. In high school (Grade 9 and OSSLT), the risk shifts to the student where 
there is greater potential for collusion and other forms of cheating. Yet, rather 
than using a varied quality assurance approach that considers the unique risks 
associated with each assessment, the audit team found that the Agency used 
substantially the same approach for all assessments. 


The audit team learned of British Columbia’s approach to potential cheating by its 
students. It initiated a formal complaints process that outlines the responsibilities 
of students, schools, and school boards, and includes the completion of 
standardized forms describing each incident and the actions taken. The Auditor | 
also noted that some jurisdictions, such as Massachusetts, may go so far as 
revoking a teaching license if a teacher is found to have violated testing policies. 


The Auditor recommended that the Office enhance its quality assurance 
procedures by implementing a formal complaints process to help determine 
whether there are any trends. As a further deterrence to non-compliance with 
assessment guidelines, the Office should consider more complete disclosure when 
test results at a particular school are withheld. Lastly, the Auditor recommended 
that the Office tailor its quality assurance processes to address the unique risks 
associated with each of the different assessments (primary, junior, high school). 


The Office takes complaints regarding non-compliance with assessment 
guidelines very seriously and has rigorous quality assurance processes to ensure 
that the administration of the assessment is consistent across the province. 
Furthermore, the Office follows up complaints at the school and board level, and 
in 2009 introduced a standardized format for such investigations. Now, the Office 
has a clear protocol for investigating and withholding results when warranted. 
Specifically, the EQAO informs parents and the public when schools’ results are 
withheld by noting on the school’s report on the EQAO website and on the 
Individual Student Report (ISR) that there has been an irregularity. In addition, a 
_ letter is attached to the ISR report to advise the parents of those students who took 
the test. 


In September 2010, subsequent to the Standing Committee’s public hearings on 
the EQAO, the press reported that ten public schools in Ontario have been 
investigated for possible cheating and irregularities on last year’s province-wide 
EQAO assessment. As a result, the Office has withheld scores for all 10 of these 
schools pending completion of its probe. The press articles cited examples of 
some of the irregularities. It is alleged that some educators engaged in cheating, 
while others inadvertently broke the rules. The chief assessment officer at the 
Education Quality and Accountability Office told the Globe and Mail that it 
learned of the problems from parents or school officials.” 


In the wake of the press articles, the Agency publicly indicated plans to 
supplement its existing teacher’s manual with a checklist on how to administer the 
test in the classroom. For example, the checklist would include whether a 
calculator is allowed on the math test. Further, it would inform teachers to not 
read passages aloud on the reading test. According to Agency officials, the 
instructions will be made more explicit and clear. ' 


8’ Ontario, Ministry of Education, Office of the Deputy Minister, S. 3.04 Education Quality and 
Accountability Office, Summary Status Table (March 2010), p. 2. 

° See Kate Hammer and Karen Howlett, Cheating probe ensnares educators, Globe and Mail, 
September 21, 2010, p. Al. 

'° Caroline Alphonso and Karen Howlett, “Ontario clarifies standardized-testing rules to-weed out 
cheaters,” Globe and Mail, September 26, 2010. 


Committee Recommendations 
The Standing Committee on Public Accounts recommends that: 


3. On the matter of irregularities and cheating involving last year’s 
EQAO assessments (confirmed in the press by the Education Quality 
and Accountability Office), the Committee asks that the Office report 
to the Standing Committee summarizing the key findings or 
conclusions from the investigations and outline any resulting policy or 
procedural changes it plans to implement to help deter or combat 
such irregularities in future. 


4. The Ontario College of Teachers tell the Standing Committee on 
Public Accounts what sanctions, if any, College officials would 
recommend be applied to teachers who are found to have violated 
EQAO testing rules. 


Large Fluctuations in EQAO results 


The audit team noted that some schools’ EQAO results fluctuated by as much as 
50% from one year to the next. While an improvement of this magnitude could 
raise suspicion, such significant swings could be caused by many legitimate 
factors. Yet, there was no systematic Agency follow-up of such cases to 
determine what accounted for such a dramatic change and whether any 
intervention is required. 


The Auditor recommended that the Agency enhance its quality assurance 
procedures by investigating any abnormally large variations in school assessment 
results from year to year and determining whether or not they are justified. 


Public Hearings 


Members asked the witnesses why the Office does not undertake a formal analysis 
or investigation to determine the cause of sudden or significant fluctuations in 
schools’ EQAO assessment results. 


- Agency officials responded that the Office had a process for reviewing such 
fluctuations (up or down) among the 72 school boards. The Office CEO typically 
telephoned the director of education for the board in question to ask about the 
reasonableness of the results, ultimately examining the schools that contributed to 
the unusual upswing or downswing. Where there was a concern, such as a 
dramatic rise in scores, the Office would begin an investigation. In such cases the 
school board superintendent would become involved. Going forward, the Office 
will require the superintendent to submit a written report describing the 
investigation and justifying his or her opinion why the results are valid. 


Members asked the Auditor whether the process as described by the CEO 
constituted the formal process that the Auditor had been seeking. The Auditor 
responded that the steps taken by the EQAO to follow up had not been formally 


documented and should be. Moreover, while obtaining board feedback was 
useful, the Auditor also suggested that EQAO officials should consider contacting 
the school in question and ask the principal for a written explanation of the 
unusual fluctuation. 


Committee Recommendation 
The Standing Committee on Public Accounts recommends that: 


5. The Education Quality and Accountability Office report to the 
Standing Committee whether it has now adopted a formal process to 
investigate and document the steps taken by the Office when its 
officials become aware of unusual fluctuations:in the assessment 
results. The Office should also report to the Committee whether it 
might consider adding a section to the standard School Reports issued 
by the Office to every elementary and secondary school in Ontario. 
This section would provide an explanation for the incidence of 
unusual fluctuations in the school’s assessment results where these 
have occurred. 


Use of Grade 9 Math Results 


Ministry of Education policy states that 30% of a student’s final grade will be 
based upon a final evaluation which may include an end-of-course exam. 
Consistent with this policy, and to help motivate Grade 9 applied math students 
who consistently fall short of the provincial standard, schools are allowed to 
incorporate the scores from the EQAO Grade 9 Assessment of Mathematics into a 
student’s end-of-course mark. However, in the 2008/09 school year, the audit 
team found these results being applied inconsistently throughout the province. 
EQAO scores accounted for anywhere from zero to 15% of a student’s final mark 
at the schools visited by the Auditor. 


The Auditor recommended that the Office review Grade 9 applied mathematics 
results to assess whether incorporating EQAO results into the student’s final mark 
is effective in motivating students and, if so, consider adopting a more consistent 
approach. 


The Office agrees with the recommendation that the practice of applying EQAO 
results to Grade 9 end-of-course marks be reviewed. Officials reported having 
included survey questions on the 2009-10 Grade 9 teacher and student 
questionnaires to determine the extent to which teachers use the EQAO tests as 
part of the student final term mark. The Office will share this information with the 
Ministry for the purpose of assessing the best course of action. 


Public Hearings 


It is generally perceived that students tend to take more seriously those 
evaluations that count in their final term mark, and less seriously those that do 
not. Therefore, Members asked the witnesses whether the students might take the 


10 


EQAO Grade 9 math assessments more seriously if it was generally known that 
some portion of it were to be included in their final grade. Members asked 
whether the Ministry is planning to analyze this possibility further. 


Agency officials responded that using or not using parts of the EQAO test as 
contributing to that final term mark is a local policy—either a decision of the 
school or of the board. The Office has, therefore, included the following question 
to teachers in its survey: “Are you using any part of this EQAO assessment as part 
of your term grade?” The purpose of this survey question is to determine whether 
using the assessment as a part of the final term mark influences student 
achievement on the test, particularly for those in Grade 9 applied math. 


Supplementary Information | 


The detailed provincial results of the EQAO 2009-10 student survey about 
counting some or all of the Grade 9 Assessment of Mathematics as part of 
students’ class marks are attached as Appendix A. When students were asked 
whether having the Grade 9 Assessment of Mathematics count as part of the their 
final term mark motivates them to take the assessment more seriously, 69% of 
Grade 9 Applied students and 72% of Grade 9 Academic students responded that 
it did. 


Committee Recommendation 
The Standing Committee on Public Accounts recommends that: 


6. The Ministry of Education and the Education Quality and 
Accountability Office report to the Standing Committee the 
conclusions they have drawn from the published results of the EQAO 
survey questions posed to Grade 9 teachers and students that asked 
whether using the EQAO assessment scores as part of the final term . 
mark influences student achievement on the test. The Committee also 
requests that the Ministry report back as to whether it has considered 
having a prescribed minimum percentage (as well as the 30% 
maximum) of the Grade 9 Assessment of Mathematics that may count 
as part of the student final term mark. 


Reporting on EQAO Assessment Results 


The Agency must report to the public and to the Minister of Education on the 
testing results, and, generally, on the quality and effectiveness of elementary and 
secondary school education. For both English and French language students, 
EQAO assessment results for each subject area are reported by school, school 


'! Education Quality and Accountability Office EQAO, Provincial Results, Grade 9 Assessment of 
Mathematics, 2009-2010, Student Questionnaire (Toronto: The Office, September 29, 2010), pp. 
15-16, Internet sites at (applied) 


http://www.eqao.com/pdf_e/provquestionnaire/0910/G9 2010 SQap_P01.PDF and (academic) 


http://www.eqao.com/pdf_e/provquestionnaire/0910/G9 2010 SQac_ PO1.PDF accessed on 
October 22, 2010. 


School Year | 


1999/2000 
2000/01 
2001/02 
2002/03 
2003/04 
2004/05 
2005/06 
2006/07 

2007/08 
2008/09 


1] 





board, and province wide; and by gender, by English language learners, and by 
special needs students. The results are compared with prior years’ results, More 
detailed contextual results are available to schools and school boards through a 
ie website. Parents also receive an individual report detailing their child’s 
results, 


The Auditor recommended that the Office improve its policies, processes and 
procedures designed to produce accurate and reliable reports for use in improving 
student performance. 


English and French language Students 


Members drew witnesses’ attention to Figure 2 of the Auditor’s 2009 Annual 
Report, which shows.the percentage of students achieving the provincial standard 
for reading, writing, and math in Grades 3 and 6 for both the English-speaking 
and French-speaking boards. 


Figure 2: Grades 3 and 6 - Percentage of Students Achieving Provincial 


Standard (Levels 3 and 4) - 1999/2000 - 2008/09 
Source of data: EQAO 
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2009 Annual Report, OAGO, p. 132. 


Members pointed out that when one compares the Grade 3 math assessment 
averages for the English-speaking students with those same students moving on 
three years later to Grade 6, the math averages are on a downward trend.” If, 
however, one looks at the comparison with the French-speaking system for the 
same time frame, in the first year the math average rises from 41% to 66%; next it 
moves from 40% to 70% and then from 47% to 74%. Members asked the 
witnesses what might explain such a dramatic improvement among the French- 
speaking students in their math scores vis-a-vis the English-speaking students in 
the same grades over the same time frame. 





!2 For example, in 1999/00 Grade 3 English-speaking students moved to Grade 6 in 2002/03. The 
math average fell from 57% down to 53%. In the following year it moved from 61% to 57%. The 
next year it rose, 58% to 60% and then 57% to 61%. 


“French: [speaking 


Math 
57 
60 
63 


66 © 


70 
74 
76 
76 
78 
80 
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Under questioning, witnesses from the Ministry and the Education Quality and 
Accountability Office indicated the following: 


» The French language boards have had full-day learning for about 10 years and 
evidence shows that early learning does pay off in terms of children being 
ready to learn in Grade 1. 


» The French-language curriculum is a curriculum for the francophone | 
community. While the core principles are the same, the expectations of the 
French language math curriculum are different from the English language 
math curriculum. 


« The French-speaking community—although spread across the province and 
very diverse—has a smaller cohort of students and teachers and may benefit 
from more focused opportunities to bring all of the educators together. As a - 
group they have the same discussions, share the same understanding, and use 
common approaches along with similar implementation of practices. In ‘other 
words, the francophone community may have a bit of an edge in terms of 
getting their entire teacher body “‘on the same page” at the same time. 


- The French and English tests are different and they are developed separatély. 
The core principles are the same but there are differences in sequencing and 
emphasis of curriculum. The different curriculum and expectations therefore 
yield different tests and possibly results. 


« The French language schools, however, share the same features as English 
language schools. For example, the Grade 3 elementary teacher teaches 
English and math. In the secondary schools they specialize. The French 
secondary schools are very small, and often teachers will teach outside their 
specialty. But for languages and math, they try to keep those teachers teaching 
in their specialties to ensure that they provide the best programs possible. 


Committee Members expressed surprise on learning that the English-speaking and 
French-speaking curriculum and assessments differ—not only for mathematics in 
Grade 3 and 6 but for reading and writing skills as well. For example, French 
language educators in consultation with the Ministry have decided to introduce 
the poem in Grade 4 as opposed to Grade 3. By the end of Grade 8, however, 
both French-speaking and English-speaking students will have basically covered 
similar material except that it is covered at different times. 


Supplementary Information 


Subsequent to the hearings, the Standing Committee asked the Ministry and the 
Agency to summarize the key differences in the curricula and assessments 
between the French language system and the English language system for Grades 
3 and 6 in reading, writing, and mathematics. 


Ministry officials noted the key differences in the curriculum as follows: 


The English-language and French-language elementary school curricula have 
equivalent learning expectations and high standards. Both are developed 
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according to the same principles and guidelines. During curriculum review, 
Ministry of Education teams work to ensure consistency and alignment between 
the expectations in both the English and the French curriculum documents. ° 


In its curriculum, the English-language Boards use Language while the French- 
language Boards use Frangais. While oral communication, reading, and writing 
strands are the same, the two curricula differ in the contents of the strands. The 
curriculum Frangais integrates media literacy in three strands—oral 
communication, reading and writing—whereas it is a separate strand in the 
Language curriculum.” 


In Mathematics, both curriculum documents cover the same content in each grade 
level thereby allowing all Ontario Grade 8 students to acquire the same 
knowledge and skills in Mathematics at the secondary level. As with Language 
and Francais, however, the contents of the strands are organized differently. In 
Grade 3 for example, the strand Number Sense and Numeration is broken into 
four elements for French-language students and three for English-language 
students. This variation may be related to the language, the acquisition of 
vocabulary, or the need to sequence the elements slightly differently for 
pedagogical reasons.'° 


In the Ministry’s view, the English-language and French-language curricula, as 
well as the learning and teaching approaches used by teachers, reflect the needs of 
the students. Moreover, culturally-sensitive pedagogical resources, ongoing 
teacher training, and an integrated approach to teaching, constitute the main 
differences between French and English-language curriculum implementation. 
These allow for differentiation of both instruction and learning environments 
thereby contributing to students’ success. © 


The key differences in the EQAO assessments between the English language and 
French language systems are as follows: 


The EQAO tests are based on the overall expectations for the French language 
(FL) and English language (EL) systems. The structure by level of difficulty is 
the same in the FL and the EL assessments in reading, writing and mathematics. 
However, in math, the proportion of questions may vary by strand, and in 
language, by skill type. i 





'3 Correspondence from Mr. Kevin Costante, Deputy Minister, Ministry of Education (May 26, 
2010), p. 1. 

Ibid. 

'S Thid., p. 2. 


16 1h: | 
Ibid. 
'7 The information requested by the Standing Committee on the EQAO assessments, notably the 


key differences between the EL and FL systems, was provided by EQAO officials. 
Correspondence from Marguerite Jackson, Chief Executive Officer, Education Quality and 
Accountability Office (May 21, 2010). 
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The EQAO French language test is not a translation of the English test. However, 
criteria used to develop the test as well as the validation process of the test are 
equal in terms of rigour and quality.'® 


The Committee was also interested in knowing whether the FL EQAO 
assessments for Grades 3 and 6 in reading, writing, and mathematics have, in all 
cases, the same number of multiple-choice questions and open-response questions 
as the EL EQAO assessments. 


Agency officials note that the two assessment programs are identical in form. 
They contain the same number of open-response and multiple-choice questions at 
each of the different grades. The items selected reflect a defined curriculum for 
each population, and assessments follow a pattern of building test challenges on 
curriculum that teachers have been expected to cover at specific points on a 
student’s journey through school (the end of the primary division in Grade 3; the 
end of the junior division in grade 6; Grade 9 math; and the OSSLT representing 
minimum literacy expectations at the end of Grade 9 and administered in Grade 
103)" 


The population sizes and characteristics of English-language boards and French- 
language boards vary significantly. The government does not intend to compare 
English- and French-language EQAO results.”” 


Committee Recommendation 
_ The Standing Committee on Public Accounts recommends that: — 


7. Because of the differences between the Ontario English language and 
French language curriculum expectations and the related EQAO 
testing, the results are not comparable. However, these differences 
are largely unknown to the public and ought to be made clear. The 
Standing Committee seeks the views of the Ministry of Education and 
the Office as to whether this can best be accomplished through the 
public websites of the Ministry of Education and the Education 
Quality and Accountability Office, or through other public reporting. 


Gender 
Public Hearings 


Agency officials explained that individual schools receive a very detailed EQAO 
report from the Agency that provides aggregate information over time for their 
students. They also receive sub-reports on gender—how well boys and girls did 
on each curriculum expectation. 


'§ Thid. 
i |, bids P. 1. 
2 ontario. Ministry of Education, Responses to Committee’s Questions following EQAO 


presentation to the Standing Committee on Public Accounts. Correspondence from Deputy 
Minister’s Office (June 7, 2010), p. 1. 
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There is a perception that girls are superior in reading and writing but weaker in 
maths and science. One Member spoke of teachers’ concerns that the Grade 3 
assessment of reading, writing, and mathematics may be gender biased in that it 
may be more difficult for male students. He asked officials. whether that 
perception is consistent with EQAO assessment results. 


Some witnesses spoke of developing the test so as to weed out gender bias in the 
reading selections and actual test questions before they become part of the 
assessment. Ministry officials spoke of the achievement gap between female and 
male students in Ontario, and of officials visiting schools where the gap is less 
pronounced to see what the schools are doing right. For example, in one northern 
Ontario school for special needs elementary students, children were learning how. 
to engage in writing using a laptop which seemed an amenable intervention both : 
to the gender issue (boys like gadgets and technology) as well as the special 
needs. 


Referring to the other historical perception that female students tend to be weaker 
than male students in maths and science, one Member asked the witnesses if 
female students are showing improvement in their math skills. In response the 
Committee was told that through Grade 9, female students are head-to-head with 
male students in mathematics. For example, across the province in last year’s 
Grade 3 EQAO assessment, 69% of female students and 67% of male students 
met the standard on the mathematics portion of the test. Initiatives and 
interventions that started 15 to 20 years ago to encourage female students in 
maths and sciences appear to be paying off. Some educators think that similar 
initiatives might work for male students who are struggling. with reading and 
writing. 


EQAO’s Relationship with Educators—Outreach 


As noted earlier, the Office provides a series of annual provincial reports that 
include a summary of high level trends, school success stories, and strategies for 
student improvement. These reports include the results of questionnaires filled 
out by students, principals, and teachers. Many of the teachers interviewed by the 
audit team stated that they found the questionnaires repetitive from year to year; 
and that they lacked the scope for general feedback or the chance to raise other 
issues. In 2009 the Office initiated a pilot communications strategy, seeking open 
feedback from a number of school staff on the EQAO student assessment process. 


The Auditor recommended that the Office consider formalizing its pilot initiative 
to provide more open-ended questions for principals, teachers, and students to 
obtain better feedback as to any concerns, and as to ways to improve the 
assessment process, 


One of the suggestions made by the teachers’ federations is that the standard 
every-child testing in Grades 3, 6, 9 and 10 be scaled back to a random sample 
due to its cost and intrusiveness. Members asked the witnesses about the rationale 
behind the EQAO policy of every-child testing. 
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It was explained that the policy originates from the Royal Commission on 
Learning which was of the view that it was reasonable to have a check on student 
learning at a few critical transition points and that parents should know how their 
child is doing relative to a provincial standard; If only aggregate data is examined, _ 
- one would find much to celebrate. But one cannot provide detailed feedback to 
every school. It is only when the data is disaggregated that a picture emerges of 
the 30% of schoolchildren who are not achieving. Without every student’s data, 
that kind of analysis cannot be done, according to Agency officials. 


During the public hearings, Members asked witnesses how the Office seeks to 
address the public “push and shove” of some of the teachers’ federations toward 
the EQAO mandate. . 


Ministry and Agency officials responded that they take a series of actions. 
Teachers are invited to symposia where data training sessions are conducted. 
They are also invited to sit on advisory councils to examine EQAO processes and 
provide feedback. EQAO officials work hard at listening and responding to 
teachers’ concerns. For example, the Office shortened the assessments 
substantially. As well, a commitment was made to the teachers to link the tests to 
the curriculum. Officials try to put a face to the Agency. A group of five within 
the Office travels the province meeting with principals, teachers, and parents in 
order to help stakeholders see that the evidence generated by EQAO assessments 
can be helpful to them. 


Ministry officials also pointed out that thousands of teachers are engaged with the 
EQAO process through activities such as test development, administration, 
assessments, and expert panels at all levels—an extraordinary achievement, in the — 
Ministry’s view. At the same time, the Office understands that the schools belong 
to a wider group of stakeholders. Others—students and parents—deserve to have | 
confidence in Ontario’s education system. Witnesses added that they have great 
relationships with teachers; they just happen to have a different point of view on 
the value of the EQAO. 


The Committee is aware that EQAO testing continues to be a concern among 
some Ontario teachers, The Committee is also aware that in 2009, the Agency 
initiated a pilot communications strategy to obtain more open feedback from a 
number of school staff on the EQAO’s student assessment process. 


Committee Recommendations 
The Standing Committee on Public Accounts recommends that: 


8. The Education Quality and Accountability Office continue its policy 
of every-child testing. 


9. The Education Quality and Accountability Office report to the 
Standing Committee the summarized results of feedback from the 
Agency’s pilot communications strategy. 


SSS 


APPENDIX: A— EQAO GRabE 9 ASSESSMENT OF MATHEMATICS 
2009-10 STUDENT QUESTIONNAIRE: ”' — 









Grade 9 Student Questionnaire Provincial Results _ 


Applied Course Academic Course 
All Students (43,201)”” All Student (97,137)? 


10. Responses of students asked if teacher will count some or all parts of the Grade 9 


Assessment of Mathematics as part of their class mark. 


11a. Responses of students asked if they were told how much the assessment will count 
as part of their class mark. : oS ee ae | 













26-30% 





other 










12. Responses of students asked if having the Grade 9 Assessment of Mathematics 
count as part of their class mark motivates them to take the assessment more seriously. 


no 






as ao | Tal wee woRLeNt a RTT 


*'The table is adapted from the EQAO Provincial Results, Grade 9 Assessment of Mathematics, 
2009-2010, Student Questionnaires (Applied and Academic) (September 29, 2010), pp. 15-16 


Internet sites at http://www.eqao.com/pdf_e/provquestionnaire/09 10/G9_2010_SQap_PO1.PDF 
(applied) and http://www.eqao.com/pdf_e/provquestionnaire/0910/G9_2010 SQac_P01.PDF 


(academic) accessed on October 22, 2010. 

22 \iumbers and percentages for Questions 11a and 12 are based on the number of Applied math 
students who answered “‘yes” to Question no. 10 (16,297). Note: Numbers and percentages for 
Question 11b are further based on the number of students who answered “yes” to Question | la. 
(14,059). See Ibid. (Applied) pp. 6-7. 

23 Numbers and percentages for Questions 11a. and 12 are based on the number of Academic math 
students who answered ‘“‘yes” to Question no, 10 (62,371). Note: Numbers and percentages for 
Question 1 1b are further based on the number of students who answered “yes” to Question | la. 
(57,052). See Ibid. (Academic) pp. 7-8. 
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CONSOLIDATED LIST OF RECOMMENDATIONS 


_ The Standing Committee on Public Accounts asks the Ministry of Education and 
the Education Quality and Accountability Office (EQAO) to provide the 
Committee Clerk with a written response to the Committee’s nine 
recommendations within 120 calendar days of the tabling of this report with the 
Speaker of the Legislative Assembly. 


The Standing Committee on Public Accounts recommends that: 


1. 


On the matter of principals exempting students fromi EQAO assessments 
for acceptable reasons and then assessing such students as though they did 
not achieve the provincial standard, the Standing Committee asks that the 
Ministry of Education and the Education Quality and Accountability 
Office report to the Standing Committee as to whether they have 
considered other options, in particular, for those schools where a 
disproportionately large number of students have been exempted. 


. The Education Quality and Accountability Office report to the Standing 


Committee the numbers of exempted students over the past five years with 
an explanation for any trends that have occurred over that timeframe. 


. On the matter of irregularities and cheating involving last year’s EQAO © 


assessments (confirmed in the press by the Education Quality and 
Accountability Office), the Committee asks that the Office report to the 
Standing Committee summarizing the key findings or conclusions from 
the investigations and outline any resulting policy or procedural changes it 
plans to implement to help deter or combat such irregularities in future. 


The Ontario College of Teachers tell the Standing Committee on Public 
Accounts what sanctions, if any, College officials would recommend be 
applied to teachers who are found to have violated EQAO testing rules. 


The Education Quality and Accountability Office report to the Standing 
Committee whether it has now adopted a formal process to investigate and 
document the steps taken by the Office when its officials become aware of 
unusual fluctuations in the assessment results. The. Office should also 
report to the Committee whether it might consider adding a section to the 
standard School Reports issued by the Office to every elementary and 
secondary school in Ontario. This section would provide an explanation 
for the incidence of unusual fluctuations in the school’s assessment results 
where these have occurred. 


. The Ministry of Education and the Education Quality and Accountability 


Office report to the Standing Committee the conclusions they have drawn 
from the published results of the EQAO survey questions posed to Grade 9 
teachers and students that asked whether using the EQAO assessment 
scores as part of the final term mark influences student achievement on the 
test. The Committee also requests that the Ministry report back as to 
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whether it has considered having a prescribed minimum percentage (as 
well as the 30% maximum) of the Grade 9 Assessment of Mathematics 
that may count as part of the student final term mark. 


. Because of the differences between the Ontario English language and 
French language curriculum expectations and the related EQAO testing, 
the results are not comparable. However, these differences are largely 
unknown to the public and ought to be made clear. The Standing | 
Committee seeks the views of the Ministry of Education and the Office as 
to whether this can best be accomplished through the public websites of 
the Ministry of Education and the Education Quality and Accountability 
Office, or through other public reporting. 


. The Education Quality and Accountability Office continue its poly of 
every-child testing. 


. The Education Quality and eect Office report to the Standing 
Committee the summarized results of feedback from the Agency’s pilot 
communications strategy. 
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